Classifying Learner Engagement through Integration of Multiple Data Sources

Intelligent tutoring systems (ITS) can provide effective instruction, but learners do not always use such systems effectively. In the present study, high school students’ action sequences with a mathematics ITS were machineclassified into five finite-state machines indicating guessing strategies, appropriate help use, and independent problem solving; over 90% of problem events were categorized. Students were grouped via cluster analyses based on self reports of motivation. Motivation grouping predicted ITS strategic approach better than prior math achievement (as rated by classroom teachers). Learners who reported being disengaged in math were most likely to exhibit appropriate help use while working with the ITS, relative to average and high motivation learners. The results indicate that learners can readily report their motivation state and that these data predict how learners interact with the ITS. Learner motivation & tutoring systems Technology-based instruction is becoming an important resource to improve learning outcomes in K-12 classrooms. Intelligent tutoring systems have been shown to improve learner achievement when used for supplemental instruction in the classroom (e.g., Koedinger, Corbett, Ritter, & Shapiro, 2000). Traditionally, tutoring systems have focused primarily on tracing students’ knowledge states. For example, the Cognitive Tutor identifies and responds to misconceptions in students’ solutions for algebra and geometry problems. However, there is growing recognition that student motivation and engagement must also be considered in addition to cognitive processes. Specifically, learners often do not use tutoring systems effectively. For example, in the case of mathematics tutoring systems, learners may choose random answers (guess), repeatedly request help until the correct answer is revealed (help abuse), or skip problems (avoidance). Recent work has focused on the goal of attempting to estimate the probability that the learner is disengaged or is “gaming” the tutoring system by using time traces of student actions with the ITS (cf., Aleven & Koedinger, 2000; Beck, 2005). The results suggest that additional _______ Copyright 2006, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. data sources may be useful in order to improve the ability of an ITS to diagnose the learner’s goals: to use the system to learn, to solve the problems independently, or to game the system. More specifically, this approach has the potential to identify when students are disengaged, but does not help us to understand why, in terms of the beliefs and goals that individual students bring to the learning situation. Different students may appear disengaged for different reasons: One may act bored because he genuinely finds the work too easy; another may be capable of doing the work but lacks confidence and feels too anxious about failure to concentrate; and another may not have the required skills but is wary of using the ITS help because she has learned not to expect useful assistance from peers, parents or even teachers. It is unlikely that a single pedagogical response will be appropriate for all cases. Rather, the engagement tracing approach might well be enhanced by additional data sources about students’ domain-specific expectations and learning goals. The present research focuses on the integration of selfreport data about learners’ motivation with teacher reports of learner motivation and achievement, and classification of learner action patterns into finite-state machines. We decided to use student self-report data about motivation for two reasons: First, literally hundreds of studies in educational psychology indicate that learner motivation can be readily reported by learners, and that these data are strongly related to a cluster of behaviors associated with learner achievement, including effective self-monitoring, goal-setting, and study behaviors (for a review, cf., Schunk, 2004). In some studies, learner motivation and self-regulation are even stronger predictors of achievement than prior academic achievement or socioeconomic status (Byrnes, 2003; Zimmerman & Martinez-Pons, 1986). Thus, adding estimates of learner engagement could improve the effectiveness of tutoring systems, in terms of pedagogical decisions to restrict access to help or to force learners to view help. Our second reason to evaluate the potential of student selfreports of motivation was more pragmatic. Much promising research focuses on the use of fragile, expensive and intrusive sensors (e.g., eye-tracking, skin conductivity,